Quantum Phase Transitions in the Hubbard Model on Triangular Lattice 
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We investigate the quantum phase transitions in the half-filled Hubbard model on the triangular 
lattice by means of the path-integral renormalization group (PIRG) method with a new iteration 
scheme proposed recently. It is found that as the Hubbard interaction U increases, the paramagnetic 
metallic state undergoes a first-order phase transition to a nonmagnetic insulating (NMI) state at 
Uci ~ 7.7t, which is followed by another first-order transition to a 120° Neel ordered state at 
Uc2 ~ lot, where t is the transfer integral. Our results ensure the existence of the intermediate NMI 
phase, and resolve some controversial arguments on the nature of the previously proposed quantum 
phase transitions. We find that k-(BEDT-TTF)2Cu2(CN)3 is located in the NMI phase close to the 
metal-insulator transition point. 



Strongly correlated electron systems with frustration 
have attracted much interest recently. There are a 
number of intriguing phenomena that have revealed 
new aspects of electron correlations. One of the strik- 
ing examples can be found in an organic compound 
K - (BEDT - TTF)2Cu2(CN)3 [J, i, E i], for which 
the triangular lattice structure of dimerized BEDT-TTF 
molecules plays an invaluable role in stabilizing a non- 
magnetic spin- liquid insulating state down to 20mK 
The nonmagnetic insulating (NMI) state, which is to- 
tally different from the naively expected 120° Neel or- 
dered state (120Neel) [1, 0], poses an interesting and 
challenging problem in the Mott transition of strongly 
correlated electrons on the frustrated triangular lattice 
0, S, i, E, El, [13, [13, [13, [il, [11, [13, [13, [ll • in par- 
ticular, it has been a central issue to figure out whether 
such a NMI state is really stabilized on the triangular 
lattice, and if so, how we can theoretically describe the 
Mott transition without any kind of magnetic ordering. 
The answer to the question should certainly provide us 
with deeper understanding of the Mott transition with 
strong frustration. 

Low-energy properties of such frustrated organic com- 
pounds may be described by the single-band Hubbard 
model on the triangular lattice at half filling P, [3| . There 
have been a number of theoretical investigations onquan- 

turn phase transitions of the model 0, [3, [3, [13, [m, 113, 
13, [13, [13, [13, [l3, [13, [l3| • However, most of conventional 
mean-field and variational treatments fail to describe the 
NMI phase, suggesting that the system may prefer the 
120Ncel state. Among those intensive studies, the pio- 
neering work by means of the path integral renormaliza- 
tion group (PIRG) method [3| is believed to provide the 
most reliable results since it can fully incorporate quan- 
tum fluctuations on the basis of an unbiased scheme. By 
this method, Morita et al. reached the remarkable con- 
clusion that the Mott transition occurs from the metallic 
state to the NMI state [3|, and the corresponding tran- 
sition may be continuous. Although the conclusion shed 
new light on the Mott transition, there still remain some 



serious questions/points: (i) Is such a continuous tran- 
sition really possible in the fully frustrated system? (ii) 
The obtained critical value of the Hubbard interaction 
seems much smaller than the values deduced by other 
methods (see below), (iii) It is difficult to study the mag- 
netic instability to the 120Neel phase by the naive PIRG 
method, so that it is unclear whether the NMI phase 
is indeed realized against the magnetic instability. We 
note that a more recent variational-cluster study found 
the metallic, NMI and 120Ncel phases in a certain pa- 
rameter regime, but the transition points could not be 
estimated due to the problem of its numerical accuracy 
[l3 |. Therefore, it is highly desirable to precisely deter- 
mine the ground-state phase diagram and clarify the na- 
ture of the associated quantum phase transitions in order 
to elucidate the essential properties inherent in the Mott 
transition under strong frustration. 

In this paper, we investigate the quantum phase tran- 
sitions in the half-filled Hubbard model on the trian- 
gular lattice. By means of the PIRG method [13, [llj 
with an improved iteration scheme proposed recently 
[i^ l- we discuss how the NMI state competes with the 
metallic and 120Neel states. By computing the dou- 
ble occupancy, the momentum distribution function and 
the spin/charge correlation functions, we find that there 
are two successive first-order quantum phase transitions 
among the metal-NMI-120Neel phases, for which the 
transition points are determined precisely. The present 
results substantially improve the previous PIRG phase 
diagram [3], and shed light on some controversial argu- 
ments on the nature of the quantum phase transitions. 

Let us consider the single-band Hubbard model on the 
triangular lattice. For simplicity, we adopt the square lat- 
tice with some diagonal bonds as shown in Fig. [U which 
is topologically equivalent to the triangular lattice. The 
Hamiltonian we consider reads 



(1) 



where Cia {c\^) is an annihilation (creation) operator of 
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FIG. 1: (a) Triangular lattice and (b) square lattice with di- 
agonal transfers, which is topologically equivalent to the orig- 
inal lattice (a). The arrows represent the spin configuration 
for the 120Neel state and the associated indices specify three 
difi'erent sublattices. 

an electron at the ith site with spin a{—^, I) and n^o- ~ 
cl^Cia- U{> 0) is the Hubbard repulsion and t(> 0) is 
the nearest-neighbor transfer integral. 

To study the ground-state properties of the Hubbard 
model eq. (HI), we use the PIRG method developed by 
Imada group [l^, [2l[ . The method is based on a simple 
idea that the true ground-state ji/jg) is obtained by act- 
ing the imaginary-time evolution operator on an initial 
state \(f>n)- |'0g) = e^^'^\4'o) with (3 ^ oo. To approach 
the true ground-state in the PIRG method, we operate 
g-AxW Tffiiii small At on an approximate ground-state 
iteratively. This method is, in principle, independent of 
an initial state and an iterative scheme employed. How- 
ever, how to choose them are crucial to reach the correct 
ground-state within restricted numerical resources. Here 
we use a new iteration scheme proposed in our previ- 
ous paper [2^ . which is extremely efficient in perform- 
ing the PIRG calculations with a large number of ba- 
sis states. We examine various initial states deduced 
from unrestricted Hartree-Fock (UHF) solutions [2^. Be- 
sides, we further consider another class of initial states 
derived from UHF solutions in the spin-rotated frame 
where the quantization axis of spin is rotated with angles 
^^0, 6'o+27r/3, and Oq+Att/S respectively for three different 
sublattices (see Fig.[T]). Performing the PIRG calculation 
with the initial states mentioned above, we discuss the 
groimd-state properties of the Hubbard model on the tri- 
angular lattice. In this paper, we focus on the system of 
6x6 sites with periodic boundary conditions. We have 
checked in a slightly different but related 2-dimensional 
frustrated lattice model that the system with 6x6 sites 
provides reliable results on the quantum phase transi- 
tions [i^. In our PIRG calculations, we keep a large 
number of states (up to 500) as the Slater basis states, 
and fix At/U = 0.5. 

We first compute the expectation value of the dou- 
ble occupancy X]iLi("-iT"'*i)/-^- The results are shown 
in Fig. [2j The introduction of the repulsive interaction 
monotonically decreases the double occupancy, implying 
that the highly correlated metallic state is realized for 
U < Uci- Further increase in the interaction gives rise 




FIG. 2: The double occupancy as a function of U/t for 
the half-filled Hubbard model on the triangular lattice with 
N = 6 X 6 sites. Circles (crosses) represent the results for the 
ground (metastable) state. 




FIG. 3: (color online) The energies per site for the compet- 
ing states in the half-filled Hubbard model on the triangular 
lattice, where circles, triangles, and squares respectively rep- 
resent the energies for the nonmagnetic metal, the NMI and 
the 120Neel. 



to two successive discontinuities in the curve at U — Uci 
and Uc2 although the latter singularity is rather weak. 
We thus find that double first-order quantum phase tran- 
sitions occur in the system. This is also supported by the 
appearance of the two cusp singularities in the curve of 
the ground-state energy, as shown in Fig. [31 By estimat- 
ing the level crossing points of energies for the competing 
states, we determine the transition points Ud ~ 7.7t and 
C/c2 ~ lot. 

Having uncovered that both of the quantum phase 
transitions are of first order, let us now discuss the 
nature of the three distinct phases in detail. To this 
end, we calculate the momentum distribution function 
n(q) and the momentum-dependent correlation func- 
tion in the charge [spin] sector N{q) [S{q)]. These 
quantities are respectively given by the Fourier trans- 
form of the site-dependent correlation functions, {c\^Cj„)^ 
{{fiifij) - {hi){hj)) and {Si ■ Sj), where n, = '}2a^i<y^ 
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FIG. 4: The momentum distribution func- 
tion n{q), and the momentum-dependent 
charge [spin] correlation function N{q) 
[S{q)] for the half-filled Hubbard model 
on the square lattice with U/t = 6,9 
and 12. For these values of interaction, 
the system is respectively in the metallic, 
NMI and 120Neel phases. Note that the 
charge correlation shows the metallic be- 
havior N{q) ~ \q\ in (d) and the insulating 
behavior N{q) ~ \qf in (e) and (f). 



Q/3 ^la'^aiiCifi and <T is the Pauli matrix. The 
computed results are shown in Fig. ID For U < Ud, 
we find that the discontinuity exists in the momentum 
distribution function at the Fermi surface (Fig. 0] (a)). 
In this case, no singularity appears in the charge and 
spin correlation functions in Figs. S] (d) and (g). We 
thus confirm that the ordinary paramagnetic metallic 
state is stabilized for U < Ud- On the other hand, 
when U > Ud, the jump singularity disappears in the 
momentum distribution function, as shown in Figs. [3] 
(b) and (c), in accordance with the Mott transition at 
U = Ud- Correspondingly, the charge correlation func- 
tion N{q) changes its |q|-depcndence (small |q| region) 
from linear to quadratic. In the region Ud < U < Uc2, 
the repulsive interaction enhances spin fluctuations at 
Qpeak = (±27r/3, ±27r/3) characteristic of the 120Neel 
phase, but does not give rise to divergent behavior. 

To discuss how magnetic fluctuations induce the 
120Neel phase for U > Uc2, we plot S{qp^^^) as a func- 
tion of J7/t in Fig. [51 We find two clear jumps at Ud and 
Uc2 that signal the first-order phase transitions. It is to 
be noted that spin fluctuations are strongly enhanced for 
U > Uc2, suggesting the emergence of the 120Neel phase. 
Although we have not performed the finite-size scaling in 
terms of different system sizes here, the tendency is con- 
sistent with the well-known fact for the triangular-lattice 
Heisenberg model 0, These observations naturally 
lead us to the conclusion that 120Ncel phase is realized 
for U > Uc2- On the other hand, in the intermediate 
phase Ud < U < Uc2, the long-range order for the 120° 
spin configuration is destroyed, where S^Qp^g^j^) does not 



show significant size-dependence. Therefore, we can say 
that the NMI state is indeed realized between the metal- 
lic and 120Neel phases. 
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FIG. 5: The spin correlation function S{qp^^,.) as a function 
of on the N = QxQ lattice at half filling for q^^ak- Circles 
(crosses) represent the results for the ground (metastable) 
state. 



It should be noted that the present results are in stark 
contrast to the previous PIRG results of Morita et al. . 
First, our numerical data clearly demonstrate the first- 
order transition between the metal and insulator, while 
Morita et al. claimed that the transition may be con- 
tinuous. Furthermore, the transition point Ud ^ 7.7t 
largely exceeds Uc ^ 5.2t obtained in the latter work 
0. Secondly, our new PIRG algorithm can describe the 
120Ncel insulating phase which could not be treated in 
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the previous study. Here we argue what really causes the 
difference between these PIRG results. In general, it may 
be difficult to discuss first-order transitions by iterative 
schemes since several competing states should be treated 
precisely on an equal footing. We find that the PIRG cal- 
culation with a naive iteration scheme may fail to reach 
the correct ground-state, since the iteration procedure 
suffers from metastable states around the metal-insulator 
transition point. We show a typical example in Figs. [2] 
and[5l Suppose we regard such a metastable state as the 
ground-state, the cusp singularity without discontinuity 
might appear in the curve of the double occupancy, as 
shown in Fig. [21 This would cause a misleading conclu- 
sion that the continuous Mott transition occurs around 
a small critical value Uc ^ 5i, which is analogous to the 
statement of Ref. Q . Our new iteration scheme can cope 
with such metastable states properly giving a clear 
description of the first-order phase transition at U = Uci 
in Figs. [21 El and m This is also the case for another phase 
transition to the 120Neel phase, for which further elab- 
orated calculations have been performed in terms of the 
spin-rotated basis states mentioned above. Consequently 
we find the first-order transition at t/ = Uc2 ^ lOi. 

metal NMI 120Neel 



FIG. 6: Phase diagram for the half-filled Hubbard model 
on the triangular lattice. Both of the transitions are of first 
order. 

We show our PIRG phase diagram in Fig. [SI which sub- 
stantially improves the previous one Q and thus sheds 
light on the controversial arguments on the nature of the 
quantum phase transitions: The metal- insulator transi- 
tion is not continuous but of first order, and the corre- 
sponding transition point Ud ~ 7.7t is much larger than 
the previous one Uci ~ 5.2; The NMI phase proposed in 
Ref. is indeed realized against the magnetic instability 
to the I20Neel phase. 

Before concluding the paper, we briefly comment on 
the nonmagnetic insulating state found in the organic 
compound k-(BEDT-TTF)2Cu2(CN)3. According to the 
band structure calculation pL], the transfer integral and 
the Coulomb interaction in the compound are estimated 
as t ~ 54.5 meV, t' ~ 57.5 meV, and U ~ 448 meV 
{U/t 8.2 and t'/t ~ 1.06), so that the system is well 
described by the isotropic triangular lattice model. By 
exploiting these values, we conclude that the above com- 
pound with U/t 8.2 is indeed in the NMI phase. In- 
terestingly, the phase diagram in Fig. [HI tells us that the 
compound is located in the NMI phase close to the first- 
order Mott transition point. 

In summary, we have obtained the ground-state phase 
diagram of the Hubbard model on the triangular lattice 



by means of the PIRG method with the improved it- 
eration scheme. Our analysis has uncovered that there 
indeed exists the NMI phase between the metallic and 
I20Neel phases. The phase diagram obtained has re- 
solved some apparently controversial conclusions on this 
issue. 
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